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[10191/3651] 
METHOD FOR DETECTING A PERSON IN A SPACE 

FIELD OF THE INVENTION 

The present invention is directed to a method for detecting a 
person in a space . 

5 BACKGROUND INFORMATION 

The method according to the present invention for detecting a 
person in a space may have the advantage of being a compromise 
between complexity and simplicity: Complex situations, such as 
persons in unusual poses, with unusual clothes, or persons 
10 that are partially concealed, are processed with little 
effort. In this connection, it should be noted that the 
"detection of a person" is understood to also include the 
detection of specific body parts. 

15 During the search, the presented method concentrates on 
comparatively few body parts that can be detected in a 
reliable and fast manner, such as the head. If the head of an 
occupant is in an unusual position in the motor vehicle, or if 
there are two occupants per seat in the motor vehicle, or if 

2 0 there exist partial concealments of the body, the respective 
heads are still found because there is no need to bring a 
complete human model into line with the scene. 

The method provides different sub-models for a body part, the 
25 different sub-models being a function of the method in the 
case of different clothing and headgear. 

Therefore, the method of the present invention is superior 
both to the model -free method and to a complex, complete CAD 
30 model because it can also cope with atypical scenarios. 

"Model-free" means that here the volume distribution, or other 
characteristics such as the curvature or texture, are directly 
evaluated by a classifier. A serious disadvantage of this 
method is that not all situations (such as the pose or 
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clothing) can be covered in the training. This results in poor 
detection performance for model -free approaches. 

Unlike the CAD model, the method according to the present 
5 invention is fast because only a few parameters are estimated. 
For a fast algorithm, rather than all measured values, it may 
be that only those of the volume to be examined may have to be 
taken into account to adapt a sub-model . The method according 
to the present invention has the characteristic that a 
10 hierarchical model of a human is adapted to the spatial data 
of a depth sensor. This depth sensor, for example, has high 
resolution. For example, the organization of the model and the 
method of adaptation to the spatial data of the depth sensor 
may be advantageous. In this context, the model may be a shape 
• 15 model (surface model) of the human body. The model may be 

designed to be articulatable , i.e., movable, and to be able to 
describe different forms of bodies and clothing. 

According to an example embodiment, the complete model may be 

2 0 adapted to follow the person over time by further adapting the 

sub-models using the data at predetermined intervals. This 
allows a so-called "tracking"; a complete adaptation no longer 
being needed, but the detected body parts being trackable in 
space and time. 

25 

According to an example embodiment, at least one body part, 
such as the head, the shoulder, or also, for example, the 
chest of a human, is of importance for the classification of 
an occupant, especially in automotive applications such as 

3 0 restraint systems or occupant condition monitoring. Moreover, 

these body parts may be easy to recognize. 

Moreover, it may be beneficial that intensity information from 
the data is used, the intensity information being texture 
3 5 characteristics, such as gray levels of the gray image or 

color values of the color image, which are each determined by 
an imaging sensor. These texture characteristics may then be 
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associated with individual body parts in order to associate 
more detailed information with the individual body parts. This 
also allows consistency checking to be able to detect a 
person . 

5 

However, the general consistency check may be performed using 
the position parameters that define the relationship between 
the individual sub-models associated with the different body 
parts. This consistency check allows the adaptation of the 
10 sub-models by the measured sensor values to lead to reasonable 
values changes . 

Moreover, the complete model may be transmitted for occupant 
classification to a restraint system in a vehicle in which the 
15 person is located. This then allows, for example, detection of 
persons; the system primarily detecting the posture of the 
occupant to identify a dangerous position of the occupant with 
respect to air bags and belt pretensioners . Thus, the 
so-called "out-of -position positions" are identified to 

2 0 prevent activation of the air bags in such positions in order 

to avoid injuries. The so-called "tracking" allows dynamic 
out-of -position sensing. Thus, the posture of the occupant is 
detected in real time. This may play a role in the case of 
fast movements of the head relative to the sensor, which are 
25 induced, for example, by vehicle decelerations during braking 
maneuvers or collisions. Here, it is also possible to classify 
the occupants based on size or biometric conditions, such as 
the average head radius or the shoulder width. 

3 0 Another use of the method according to the present invention 

is the ant i -pinch protection for extremities or objects in 
power windows. Extremity detection can also be used for 
gesture recognition. Monitoring the passenger compartment 
while the vehicle is parked may also be an application of the 
3 5 method according to an example embodiment of the present 
invention. This may be used, for example, in a car alarm 
system. It is also possible to automatically operate the child 
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security door lock against accidental opening if a child has 
been detected by the method according to the present 
invention. In combination with a temperature sensor, it is 
possible to adjust the temperature by opening and closing the 
5 windows, or by controlling the air conditioning, in case a 
person or, for example, a child is alone in the vehicle. 
Convenience features may also benefit from an example 
embodiment of the method according to the present invention. 
The adjustment of headrests, mirrors, sun visors, steering 

10 wheel and/or contoured seats can be carried out automatically. 
The control of other systems that require the exact position 
of the head is rendered possible, inter alia, a hands-free 
kit, head-up displays, or noise control systems. Other 
methods, for example, for identifying the occupants, or eye 

15 localization are aided by a precise localization of the head, 
if necessary, along with an orientation measurement. 

The depth sensor takes the form, for example, of an image 
pickup. Above all, a video sensor, such as a stereo camera 
2 0 system or a multi -camera system, may be used here. Also usable 
here are a distance-measuring matrix sensor using the 
transit-time principle or the phase -measuring principle, a 
laser scanner, a laser radar, or structured lighting. Further, 
the distance-measuring matrix sensor may use a microwave 

2 5 radar, or the measuring principle may be based on the emission 

of sound waves (ultrasound) , or an arbitrary electromagnetic 
radiation (visible light, invisible light, microwaves). 

Likewise, the representation of the sub-models contained in 

3 0 the modules can take completely different forms, such as CAD 

data, measurement data, triangulated data, superquadrics , or 
two-dimensional descriptions . 

BRIEF DESCRIPTION OF THE DRAWINGS 
3 5 Figure 1 shows a block diagram of an example embodiment of a 
device according to the present invention. 



NY01 702166 v1 



4 



SUBSTITUTE SPECIFICATION 



Figure 2 is a first flow chart of an example embodiment of the 
method according to the present invention. 

Figure 3 shows a block diagram of an example embodiment of the 
5 method according to the present invention. 

Figure 4 is a representation of an example embodiment of the 
hierarchical models . 

10 Figure 5 shows a shape model of the head in different 

variations and depths of detail, according to an example 
embodiment of the present invention. 

Figure 6 is a second flow chart of an example embodiment of 
15 the method according to the present invention. 

Figure 7 shows different mounting positions to cover the rear 
passenger compartment, according to an example embodiment of 
the present invention. 

20 

DETAILED DESCRIPTION 

Systems for sensing the passenger compartment of a motor 
vehicle are based on very different physical measuring 
principles. For example, it is possible to use a weight mat 

25 here; the pressure profile being used to determine whether the 
seat is empty or occupied. Alternatively, it is possible to 
carry out a classification. However, the sensor systems 
relevant to the method according' to the present invention are 
depth-measuring sensor systems. Here, the distance -measuring 

3 0 matrix sensor, for example, takes the form of an image pickup. 

Such an image pickup may be a video sensor, a radar sensor, an 
ultrasound sensor, or a multi-camera system. In this 
connection, it is possible to use different measuring 
principles: the transit-time principle, the phase -measuring 

35 principle, the principle of structured lighting, and the laser 
scanner principle. 
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The spatial data of the depth sensor, which generally takes 
the form of a matrix sensor and produces more than a hundred 
measured values virtually simultaneously, is used to detect a 
person in a space. In the case of the passenger compartment of 
5 a motor vehicle this is, for example, the occupant -presence 

situation. Besides the pure occupant presence, it may also be 
necessary to classify the occupants, i.e., size, pose, and 
weight. In this connection, especially the localization of 
individual body parts, such as the head or the chest, is of 
10 interest for the control of, for example, restraint systems. 
Such information is also useful for various convenience 
features, such as the adjustment of rearview mirrors. 

Figure 1 shows an example embodiment of a device according to 
15 the present invention in a block diagram. A person 6 is 
detected by a depth sensor 1 in terms of his/her spatial 
extent. Depth sensor 1 is mounted, for example, in a passenger 
compartment to act as an image pickup. The spatial data are 
transmitted to a processor 2 on which the method according to 
2 0 the present invention is executed. A restraint system 3, an 
anti-pinch function 4, and convenience features 5 are 
controlled by processor 2 as a function of the result of the 
method according to the present invention. Therefore, 
processor 2 may be located in a control unit. 

25 

Further vehicle functions are also controllable by processor 2 
so that these vehicle functions may also benefit from the 
inventive method for detecting a person. The method according 
to the present invention is applicable outside of motor 
30 vehicles, for example, for video surveillance. Access systems 
may also utilize the method according to the present 
invention. 

Figure 2 shows an example embodiment of the sequence of the 
35 inventive method in a first flow chart. In method step 6, 
sensor 1 produces sensor values that provide spatial data 
about person 6. This data is transmitted to processor 2. 
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Processor 2 loads on its memory a model of a human which is 
hierarchically structured; i.e., there exist sub-models for 
different body parts, the sub-models in turn being divided 
into further sub-models. This subdivision is done to allow 
5 optimal adaptation for different typical clothing situations 
of the particular body. Then, processor 2 selects the 
appropriate sub-model at the lowest level of the sub-models, 
and performs an adaptation using the spatial data. For this 
purpose, any optimization method may be employed. 

10 

Method step 8 checks whether this adaptation is consistent 
with respect to the position parameters. If this not the case, 
then the method returns to step 7 to repeat the adaptation. 
However, if this is the case, then, in method step 9, the 
15 complete model is gradually assembled from the sub-models. If, 
in the process, certain body parts are not found, for example, 
because of a concealment effect, a consistent interpretation 
of the spatial data is still obtained. This complete model is 
then used to recognize the person. This data is then made 

2 0 available to other functions, such as the restraint system, 

the convenience feature, or the ant i -pinch protection. 

Figure 3 shows an example embodiment of the inventive method 
in a block diagram. A person 11 is seated on a vehicle seat 
25 10, and is spatially detected by a depth sensor 12. This depth 
sensor 12 takes the form of a video sensor here, and is 
located above the windshield. However, depth sensor 12 can 
also be installed at other locations in the passenger 
compartment. The spatial measured values of sensor 12 are 

3 0 transmitted to a processor 13 which adapts the spatial model 

to the spatial measured values using the sub-models. To this 
end, a complete model 14 is loaded from a memory. Processor 13 
delivers the size, position, and posture of the occupant as an 
output quantity 15. 

35 

In Figure 4, the hierarchical model is shown by way of an 
example embodiment. The complete model is referred to as 
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"shape model 0 16". This model 16 is further divided into one 
or more units 17 through 19, which are referred to as shape 
model 1 through shape model 3. The permitted relationships 
between these sub-models, which are defined by the relative 
5 position parameters, are stored for these sub-models. 
Analogously to the division of shape model 0 into its 
sub-models, it is possible to break down these sub-models into 
sub-models as well, and to specify the relationships of the 
sub-models. The sub-models of shape model x are referred to as 

10 "shape model x.y" . In this connection, y denotes a non-zero 
natural number. The model may contain an arbitrary number of 
such levels. The number of the further subdivisions and the 
depth of the subdivisions, i.e., the number of levels, are not 
predetermined and may be handled differently for different 

15 sub-models. All branches of the model have in common that they 
contain one or more surface models as the last level. The set 
of surface models describes the respective body part in 
different forms and depths of detail. The shape model 1.17 
sub-model is composed of sub-models for head 27. Shape model 

2 0 2.18 is divided into shape models 2.1.2 0 and 2.2.21 of which 

shape model 2.1 pertains to upper arm 28 while shape model 2.2 
pertains to lower arm 29. Shape model 3.19 is divided into 
three shape models 3.1.22, 3.2.23 and 3.3.24, which are 
defined by the relative position parameters with respect to 

25 each other. Shape model 3.1.22 is divided into shape models 

3.1.1 and 3.1.2, which are provided with reference numerals 25 
and 26, respectively. Shape model 3.1.1 is divided into 
further sub-models 3 0 that define the shoulder here. Shape 
model 3.1.2 defines different pieces of torso information. 

30 Likewise, body parts 32 and 33 are associated with shape 
models 3.2.2 3 and 3.3.24, respectively. The number of the 
further subdivisions and the depth of the subdivisions, i.e., 
the number of levels, are not predetermined and may be handled 
differently for different sub-models. All branches of the 

3 5 model have in common that they contain one or more surface 

models as the last level. The set of surface models describes 
the respective body part in different forms and depths of 
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detail. During the adaptation of the model to the scene, the 
model is processed starting from certain shape models or 
individual surface models. These individual parts are 
assembled to form the complete model, taking into account the 
5 consistency with respect to the position parameters. Once the 
rough position parameters have been determined, detailed 
surfaces may be further adapted, taking into account the 
consistency conditions. 

10 If certain body parts are not detected, for example, because 
of concealment effects, a consistent interpretation of the 
spatial measured data is still obtained. 

An exemplary set of surface models of the head, according to 
15 an example embodiment, is shown in Figure 5. The complexity of 
the model variants increases on abscissa 34. An increasing 
depth of detail of the model is described on ordinate 35. The 
simplest model 3 6 becomes the more complex sub-model 37, which 
is sub-divided into a more detailed model 38 and an even more 
20 detailed model 40; the more detailed model 40 already 

representing the head shape fairly accurately. For model 38, 
there exist model 3 9 which has the same depth of detail, but 
greater complexity. Selecting a greater depth of detail for 
model 3 9 leads to model 41, which shows a head with a hat that 
25 has a brim. Sub-models 40 and 41 are ultimately used for 
adaptation to the spatial data. 

Figure 6 shows a second flow chart of an example embodiment of 
the method according to the present invention. In method step 

30 43, depth sensor 1 produces the spatial data. Method step 45 
defines a region to be examined. This is done using data of 
already processed shape models: the detection status, 
position, and position parameters from method step 44. In 
method steps 46 and 47, the model is adapted to the spatial 

35 data, taking into account the spatial shape model from method 
step 42. This step may be carried out, for example, using a 
standard method of model adaptation in terms of the minimum 
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square error. 

If this adaptation was successful, then, in method step 48, 
the detection status is stored in terms of position and 
5 position parameters. In method step 49, the position and 
position parameters of the model are then checked for 
consistency. To this end, the data of already processed shape 
models from method step 44 is reused. In method step 50, the 
results of the adaptation are recorded in the detection status 

10 of the sub - mode 1 . The detection status and the position 

parameters are passed on to the next higher level. If, again, 
relationships exist with another model, then a consistency 
check is performed until the complete model has been 
assembled. Once the model has been adapted, a subsequent, 

15 complete adaptation is no longer needed, and the detected body 
parts can be tracked in space and time. For fast and stable 
implementation, only body parts that can be detected quickly 
and reliably should be included in the model, typically the 
head, torso, and arms. The head module and the upper part of 

2 0 the body play a central role in the model because the human 

head is of outstanding importance to the set objective and, in 
addition, is comparatively invariant in shape. Moreover, the 
head is usually spatially exposed, and can therefore be 
located relatively reliably. The presented method is generally 

25 independent of the source of the spatial data. 

Figure 7, image (a) , shows an additional sensor for covering 
the rear passenger compartment, according to an example 
embodiment of the present invention. Figure 7b, in contrast, 

30 shows an alternative mounting position of the sensor for 

covering the entire rear passenger compartment, according to 
an example embodiment of the present invention. This is 
denoted by reference numeral 52, whereas in Figure 7a, sensor 
53 is provided for the front seat and sensor 51 for the rear 

35 seat. 
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ABSTRACT OF THE DISCLOSURE 
A method for detecting a person in a space is disclosed, 
including at least one depth sensor producing spatial data 
about the space to be monitored, at least one sub-model which 
5 is subdividable into further sub-models being used for at 
least one selected body part of a human, the spatial data 
being used to adapt the sub-models, the adaptation being 
checked by position parameters between the sub-models of 
different body parts, and the person being recognized using a 
10 complete model made up of the checked sub-models. 
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